Changes:

• Rename APP_ENGINE_MODE to HAS_CTYPES
• Make load/dump_session compatible with custom modules
• Add loads/dumps_session (load/dump to/from string)

Right now dump_session closes its file; it should not close it for StringIO files so that the caller can get the value of the StringIO. Unfortunately typechecking (the workaround implemented here) doesn't work as cStringIO.StringIO is actually a function, so a better workaround is needed.

I'd say if passed a file-like object that it should never close it. That should be the job of the callee.

Will also start to address this next week.

yep. I'm looking at that requested change now.

I've added the HAS_CTYPES change, and that will post soon. There seem to be two central issues remaining:

• dump_session/load_session needs to not close on StringIO, however python's doing some fake namespacing for cStringIO.StringIO. I think I have an idea that should fix that... I'm still working it out.
• more flexibility when there's a custom __main__ module

@lidavidm: that's the gist of the remaining changes, right? Can you fill me in on sympy-live's requirement again for __main__? (I mean why it's needed. -- I think ipython does similar stuff for their main in certain cases)

Regarding the close thing, I think the correct solution is if it's passed a file-like object, it should be the caller's responsibility to close it.

It's because the App Engine doesn't define __main__ http://live.sympy.org/?evaluate=__main__%0A%23--%0A

We've run into this issue with the standard library too. See http://bugs.python.org/issue16959 and https://code.google.com/p/googleappengine/issues/detail?id=8665.

This is what pickle does

In [24]: f = open('pickletest', 'bw')

In [25]: pickle.dump(1, f)

In [26]: f.closed
Out[26]: False

I'll buy the change that if a file-like object is passed in, it's the caller's responsibility to close it. That'll go in. I think there's some work to do for cStringIO.StringIO, first however.

I was pretty sure I remembered that the issue with ___main__ was that App Engine didn't use it. However, I'd expect it might be due to whatever daemon they are running is actually ___main___, and not the interpreter session... that's likely called something else, or faked with namespacing tricks like ipython does in some corner cases. I've seen wx stuff do similarly too. So, if they just don't use __name__ is __main___ for the interactive session, then I get it... and then I get what would be needed to work with that.

@asmeurer: I don't get your example, however. In standard python, you get the same behavior:

>>> __main__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name '__main__' is not defined
>>> # however this works...
>>> __name__
'__main__'

On an aside, I did notice that globals() throws an error in sympy-live.

I think there's some work to do for cStringIO.StringIO, first however.

What needs to be done?

I was pretty sure I remembered that the issue with main was that App Engine didn't use it. However, I'd expect it might be due to whatever daemon they are running is actually _main, and not the interpreter session... that's likely called something else, or faked with namespacing tricks like ipython does in some corner cases. I've seen wx stuff do similarly too. So, if they just don't use name is main_ for the interactive session, then I get it... and then I get what would be needed to work with that.

Oh, sorry, I didn't realize that __main__ has to be imported.

Anyway, I wouldn't take to heart too much the behavior of things in the SymPy Live shell, because it uses the pickling system that we are trying to get rid of with dill. The fact remains that if you import a library in the app engine that imports __main__, you get the traceback from https://code.google.com/p/googleappengine/issues/detail?id=8665.

I guess dill uses __main__ pretty extensively, so you'll probably have to figure out exactly what the situation is in the app engine.

Regarding globals() in SymPy Live, that's a bug in the printer. Change the output format to repr in the settings (the default, LaTeX, isn't very good if you are dealing with non-SymPy objects anyway).

What needs to be done with the file handler is to figure out how to deal with two things:

• pickling the open file handler
• cStringIO.StringIO objects are built dynamically, and faked to in cStringIO -- thus don't pickle correctly. I have a solution in mind that I have to finish.

I remember the reasoning behind taking a file string name as an argument instead of a file handler. It's because if you do this:

>>> f = open('foo.pkl', 'wb')
>>> dill.dump_session( f )
>>> f.close()

You'd then be pickling the open file handler, unless you do something like popping f from the session that may not be appropriate in all cases.

Ok. I'll look into what App Engine is doing with main. I expect something like I mentioned earlier... but I'll have to have a look.

Looks the daemon is __main__, while the app(s) being run are in some dummy namespace that acts like main.
https://developers.google.com/appengine/docs/python/tools/webapp/running

So it seems like the same case I've seen in wx and ipython. As long as you can get a handle to it, I think there's a generic solution. Maybe let the caller be responsible for getting the handler, as a start... that'd put me close in line with David's code.

Ahh, I didn't realize that the file object itself would be pickled because of this.

Maybe dill just needs a dump_sessions then.